Inverse Reinforcement Learning Research Articles

Data-Driven Policy Learning Methods from Biological Behavior: A Systematic Review

Policy learning enables agents to learn how to map states to actions, thus enabling adaptive and flexible behavioral generation in complex environments. Policy learning methods are fundamental to reinforcement learning techniques. However, as problem complexity and the requirement for motion flexibility increase, traditional methods that rely on manual design have revealed their limitations. Conversely, data-driven policy learning focuses on extracting strategies from biological behavioral data and aims to replicate these behaviors in real-world environments. This approach enhances the adaptability of agents to dynamic substrates. Furthermore, this approach has been extensively applied in autonomous driving, robot control, and interpretation of biological behavior. In this review, we survey developments in data-driven policy-learning algorithms over the past decade. We categorized them into the following three types according to the purpose of the method: (1) imitation learning (IL), (2) inverse reinforcement learning (IRL), and (3) causal policy learning (CPL). We describe the classification principles, methodologies, progress, and applications of each category in detail. In addition, we discuss the distinct features and practical applications of these methods. Finally, we explore the challenges these methods face and prospective directions for future research.

Evaluation method with digital expert on the criticality of car-following scenarios for autonomous vehicles testing

Evaluation of autonomous vehicles is one of the major challenges before they can be released. Due to the advantages in efficiency, cost, and safety, scenario-based simulation methods have recently received great attention. Even so, as the complexity and uncertainty exist in the real driving environment, the scenarios that autonomous vehicles may encounter are infinite. Therefore, it is necessary to classify simulation scenarios according to their criticality. It contributes to accelerating the evaluation processes. This paper presents a novel criticality evaluation method, based on a proposed Digital Expert, for car-following autonomous driving. The Digital Expert acts as the evaluator to evaluate the criticality of scenarios depending on their driving performance. Driving performance refers to the achieved degree of driving intentions. Firstly, a Digital Expert is established as the evaluator for the criticality of the scenario using the inverse reinforcement learning method. Then, based on the fact that the intention of Digital Expert is to maximize its internal reward function, the reward function is used to evaluate driving performance. Finally, calculating the criticality of the car-following scenario according to the mapping relationship between driving performance and criticality. Using the driving data in the NGSIM data set, this paper generates two groups of simulated car-following scenarios and evaluates the criticalities of the two scenarios. The experimental results show that the proposed criticality evaluation method can reasonably evaluate the criticality of car-following scenarios.

M 3 Rec: A Context-aware Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation

Reinforcement learning (RL) has shown great promise in optimizing long-term user interest in recommender systems. However, existing RL-based recommendation methods need a large number of interactions for each user to learn the recommendation policy. The challenge becomes more critical when recommending to new users who have a limited number of interactions. To that end, in this paper, we address the cold-start challenge in the RL-based recommender systems by proposing a novel context-aware offline meta-level model-based reinforcement learning approach for user adaptation. Our proposed approach learns to infer each user's preference with a user context variable that enables recommendation systems to better adapt to new users with limited contextual information. To improve adaptation efficiency, our approach learns to recover the user choice function and reward from limited contextual information through an inverse reinforcement learning method, which is used to assist the training of a meta-level recommendation agent. To avoid the need for online interaction, the proposed method is trained using historically collected offline data. Moreover, to tackle the challenge of offline policy training, we introduce a mutual information constraint between the user model and recommendation agent. Evaluation results show the superiority of our developed offline policy learning method when adapting to new users with limited contextual information. In addition, we provide a theoretical analysis of the recommendation performance bound.

SC-AIRL: Share-Critic in Adversarial Inverse Reinforcement Learning for Long-Horizon Task

SC-AIRL: Share-Critic in Adversarial Inverse Reinforcement Learning for Long-Horizon Task

Recursive logit-based meta-inverse reinforcement learning for driver-preferred route planning

Driver-preferred route planning often evaluates the quality of a planned route based on how closely it is followed by the driver. Despite decades of research in this area, there still exist nonnegligible deviations from planned routes. Recently, with the prevalence of GPS data, Inverse Reinforcement Learning (IRL) has attracted much interest due to its ability to directly learn routing patterns from GPS trajectories. However, existing IRL methods are limited in that: (1) They rely on numerical approximations to calculate the expected state visitation frequencies (SVFs), which are inaccurate and time-consuming; and (2) They ignore the fact that the coverage of GPS trajectories is skewed toward popular road segments, causing difficulties in learning from sparsely covered ones. To overcome these challenges, we propose a recursive logit-based meta-IRL approach, where (1) We use the recursive logit model to capture drivers’ route choice behavior so that the expected SVFs can be analytically derived, which substantially reduces the computational efforts; and (2) We introduce meta-parameters and employ meta-learning techniques so that the learning on sparsely covered road segments can benefit from that on popular ones. When training our IRL model, we update the rewards of road segments with the expected SVFs by solving several systems of linear equations and update the meta-parameters through a two-level optimization structure to ensure its fast adaption and versatility. We validate our approach using real GPS data in Chengdu, China. Results show that our planned routes better match actual routes compared with state-of-the-art methods including the recursive logit model, Deep-IRL and Dij-IRL: the F1-Score increases by 4.17% with the introduction of the recursive logit model and further increases to 5.19% after meta-learning is employed. Moreover, we can reduce training time by over 95%.

Recent Advancements in Inverse Reinforcement Learning

Inverse reinforcement learning (IRL) has seen significant advancements in recent years. This class of approaches aims to efficiently learn the underlying reward function that rationalizes the behavior exhibited by expert agents, often represented by humans. In contrast to mere behavioral cloning, the reconstruction of a reward function yields appealing implications, as it allows for more effective interpretability of the expert’s decisions and provides a transferable specification of the expert’s objectives for application in even different environments. Unlike the well-understood field of reinforcement learning (RL) from a theoretical perspective, IRL still grapples with limited understanding, significantly constraining its applicability. A fundamental challenge in IRL is the inherent ambiguity in selecting a reward function, given the existence of multiple candidate functions, all explaining the expert’s behavior. In this talk, I will survey three of my papers that have made notable contributions to the IRL field: “Provably Efficient Learning of Transferable Rewards”, “Towards Theoretical Understanding of Inverse Reinforcement Learning”, and “Inverse Reinforcement Learning with Sub-optimal Experts". The central innovation introduced by the first paper is a novel formulation of the IRL problem that overcomes the issue of ambiguity. IRL is reframed as the problem of learning the feasible reward set, which is the set of all rewards that can explain the expert’s behavior. This approach postpones the selection of the reward function, thereby circumventing the ambiguity issues. Furthermore, the feasible reward set exhibits convenient geometric properties that enable the development of efficient algorithms for its computation. Building on this novel formulation of IRL, the second paper addresses the problem of efficiently learning the feasible reward set when the environment and the expert’s policy are not known in advance. It introduces a novel way to assess the dissimilarity between feasible reward sets based on the Hausdorff distance and presents a new PAC (probabilistic approximately correct) framework. The most significant contribution of this paper is the introduction of the first sample complexity lower bound, which highlights the challenges inherent in the IRL problem. Deriving this lower bound necessitated the development of novel technical tools. The paper also demonstrates that when a generative model of the environment is available, a uniform sampling strategy achieves a sample complexity that matches the lower bound, up to logarithmic factors. Finally, in the third paper, the IRL problem in the presence of sub-optimal experts is investigated. Specifically, the paper assumes the availability of multiple sub-optimal experts, in addition to the expert agent, which provides additional demonstrations, associated with a known quantification of the maximum amount of sub-optimality. The paper shows that this richer information mitigates the ambiguity problem, significantly reducing the size of the feasible reward set while retaining its favorable geometric properties. Furthermore, the paper explores the associated statistical problem and derives novel lower bounds for sample complexity, along with almost matching algorithms. These selected papers represent notable advancements in IRL, contributing to the establishment of a solid theoretical foundation for IRL and extending the framework to accommodate scenarios with sub-optimal experts.

Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variables

Designing suitable reward functions for numerous interacting intelligent agents is challenging in real-world applications. Inverse reinforcement learning (IRL) in mean field games (MFGs) offers a practical framework to infer reward functions from expert demonstrations. While promising, the assumption of agent homogeneity limits the capability of existing methods to handle demonstrations with heterogeneous and unknown objectives, which are common in practice. To this end, we propose a deep latent variable MFG model and an associated IRL method. Critically, our method can infer rewards from different yet structurally similar tasks without prior knowledge about underlying contexts or modifying the MFG model itself. Our experiments, conducted on simulated scenarios and a real-world spatial taxi-ride pricing problem, demonstrate the superiority of our approach over state-of-the-art IRL methods in MFGs.

Making AI Policies Transparent to Humans through Demonstrations

Demonstrations are a powerful way of increasing the transparency of AI policies to humans. Though we can approximately model human learning from demonstrations as inverse reinforcement learning, we note that human learning can differ from algorithmic learning in key ways, e.g. humans are computationally limited and may sometimes struggle to understand all of the nuances of a demonstration. Unlike related work that provide demonstrations to humans that simply maximize information gain, I leverage concepts from the human education literature, such as the zone of proximal development and scaffolding, to show demonstrations that balance informativeness and difficulty of understanding to maximize human learning.

Data Poisoning to Fake a Nash Equilibria for Markov Games

We characterize offline data poisoning attacks on Multi-Agent Reinforcement Learning (MARL), where an attacker may change a data set in an attempt to install a (potentially fictitious) unique Markov-perfect Nash equilibrium for a two-player zero-sum Markov game. We propose the unique Nash set, namely the set of games, specified by their Q functions, with a specific joint policy being the unique Nash equilibrium. The unique Nash set is central to poisoning attacks because the attack is successful if and only if data poisoning pushes all plausible games inside it. The unique Nash set generalizes the reward polytope commonly used in inverse reinforcement learning to MARL. For zero-sum Markov games, both the inverse Nash set and the set of plausible games induced by data are polytopes in the Q function space. We exhibit a linear program to efficiently compute the optimal poisoning attack. Our work sheds light on the structure of data poisoning attacks on offline MARL, a necessary step before one can design more robust MARL algorithms.

Estimation of Different Reward Functions Latent in Trajectory Data

In recent years, inverse reinforcement learning has attracted attention as a method for estimating the intention of actions using the trajectories of various action-taking agents, including human flow data. In the context of reinforcement learning, “intention” refers to a reward function. Conventional inverse reinforcement learning assumes that all trajectories are generated from policies learned under a single reward function. However, it is natural to assume that people in a human flow act according to multiple policies. In this study, we introduce an expectation-maximization algorithm to inverse reinforcement learning, and propose a method to estimate different reward functions from the trajectories of human flow. The effectiveness of the proposed method was evaluated through a computer experiment based on human flow data collected from subjects around airport gates.

Inverse Reinforcement Learning with Agents’ Biased Exploration Based on Sub-Optimal Sequential Action Data

Inverse reinforcement learning (IRL) estimates a reward function for an agent to behave along with expert data, e.g., as human operation data. However, expert data usually have redundant parts, which decrease the agent’s performance. This study extends the IRL to sub-optimal action data, including lack and detour. The proposed method searches for new actions to determine optimal expert action data. This study adopted maze problems with sub-optimal expert action data to investigate the performance of the proposed method. The experimental results show that the proposed method finds optimal expert data better than the conventional method, and the proposed search mechanisms perform better than random search.

Estimating Objective Weights of Pareto-Optimal Policies for Multi-Objective Sequential Decision-Making

Sequential decision-making under multiple objective functions includes the problem of exhaustively searching for a Pareto-optimal policy and the problem of selecting a policy from the resulting set of Pareto-optimal policies based on the decision maker’s preferences. This paper focuses on the latter problem. In order to select a policy that reflects the decision maker’s preferences, it is necessary to order these policies, which is problematic because the decision-maker’s preferences are generally tacit knowledge. Furthermore, it is difficult to order them quantitatively. For this reason, conventional methods have mainly been used to elicit preferences through dialogue with decision-makers and through one-to-one comparisons. In contrast, this paper proposes a method based on inverse reinforcement learning to estimate the weight of each objective from the decision-making sequence. The estimated weights can be used to quantitatively evaluate the Pareto-optimal policies from the viewpoints of the decision-makers preferences. We applied the proposed method to the multi-objective reinforcement learning benchmark problem and verified its effectiveness as an elicitation method of weights for each objective function.

Special Issue on Cutting Edge of Reinforcement Learning and its Hybrid Methods

Since deep Q-networks and AlphaGO by Google DeepMind were proposed, not only a reinforcement learning integrated with deep learning but also its applications have attracted much attention on. ChatGPT, which learns by Proximal Policy Optimization as one of reinforcement learning mechanisms, is an excellent example. With this background, we propose the special issue titled “Cutting Edge of Reinforcement Learning and its Hybrid Methods,” which is the same of the special issue in 2017, to cover the latest progress and trends in reinforcement learning and its hybrid methods (combined with machine learning, neural networks, and evolutionary computation). We received 28 submissions, out of which 12 submissions were peer-reviewed and 16 submissions that were outside the scope of the special issue were excluded. After a thorough review process including comments and suggestions by reviewers, seven submissions were accepted. These included four theory and three application papers, indicating nearly equal contributions in theory and application aspects. The content of the first three papers is related to inverse reinforcement learning, while the fourth one is not directly related to reinforcement learning but it would be related to reinforcement learning in the future. All papers address issues surrounding cutting-edge technology in reinforcement learning. We would like to end by expressing that we hope and believe that this special issue can largely contribute to the development in the field of reinforcement learning while holding a wide international appeal.

How do active road users act around autonomous vehicles? An inverse reinforcement learning approach

The inevitable impact of autonomous vehicles (AV) on traffic safety is becoming a reality with the progressive deployment of these vehicles in different parts of the world. Still, many questions linger in the minds of road users that will share the road and interact with these AVs on a daily basis. To answer some of these questions, this study utilized recently collected real-world AV data from the United States, with the focus mainly targeting active road users. Specifically, the 1,492 h of recorded trips were processed to extract AV-pedestrian and AV-cyclist interactions of different movement types. The interactions were then investigated to gain a better understanding of the active road users’ behavior, while excluding any interactions that involved intervention from the AVs’ human test drivers. Through deep maximum entropy inverse reinforcement learning (DME-IRL), the reward functions describing the utility of active road users were retrieved and assessed for five interaction scenarios, including parallel, opposing, crossing, and turning (left and right) interactions. In addition, the policies developed as part of the solution were used to simulate the behavior of active road users and validate the resulting conflicts in terms of safety and evasive actions. Overall, the utilized approach demonstrated high accuracy in mimicking the interaction behavior of active road users when encountering an AV, with 81–84% accuracy in predicting evasive actions in parallel and opposing interactions and 12–17% mean absolute error for safety indicators in crossing and turning interactions. In addition, the resulting reward functions provided reliable insight onto the preferences and considerations of the active road users in these situations. Overall, cyclists tend to be less cautious around AVs compared to pedestrians, while the AVs tend to slow down and leave sufficient distance from all active road users in most cases. Still, the robotic behavior of the AVs, which can sometimes be inconsistent, leads to risky evasive actions by the active road users, which can affect other road users in busy intersections.

Integrating inverse reinforcement learning into data-driven mechanistic computational models: a novel paradigm to decode cancer cell heterogeneity

Cellular heterogeneity is a ubiquitous aspect of biology and a major obstacle to successful cancer treatment. Several techniques have emerged to quantify heterogeneity in live cells along axes including cellular migration, morphology, growth, and signaling. Crucially, these studies reveal that cellular heterogeneity is not a result of randomness or a failure in cellular control systems, but instead is a predictable aspect of multicellular systems. We hypothesize that individual cells in complex tissues can behave as reward-maximizing agents and that differences in reward perception can explain heterogeneity. In this perspective, we introduce inverse reinforcement learning as a novel approach for analyzing cellular heterogeneity. We briefly detail experimental approaches for measuring cellular heterogeneity over time and how these experiments can generate datasets consisting of cellular states and actions. Next, we show how inverse reinforcement learning can be applied to these datasets to infer how individual cells choose different actions based on heterogeneous states. Finally, we introduce potential applications of inverse reinforcement learning to three cell biology problems. Overall, we expect inverse reinforcement learning to reveal why cells behave heterogeneously and enable identification of novel treatments based on this new understanding.

Model discrepancy policy optimization for task-oriented dialogue

Task-oriented dialogue systems use deep reinforcement learning (DRL) to learn policies, and agent interaction with user models can help the agent enhance its generalization capacity. But user models frequently lack the language complexity of human interlocutors and contain generative errors, and their design biases can impair the agent’s ability to function well in certain situations. In this paper, we incorporate an evaluator based on inverse reinforcement learning into the model to determine the quality of the dialogue of user models in order to recruit high-quality user models for training. We can successfully regulate the quality of training trajectories while maintaining their diversity by constructing a sampling environment distribution to pick high-quality user models to participate in policy learning. The evaluation on the Multiwoz dataset demonstrates that it is capable of successfully improving the dialogue agents’ performance.

Underactuated MSV path following control via stable adversarial inverse reinforcement learning

Model-based control approaches are inadequate to solve the marine surface vehicle (MSV) path-following problem, especially under adverse environments. To effectively deal with the MSV path-following problem, model-free deep reinforcement learning (DRL) based methods have been developed. However, defining an efficient reward function for DRL in path following tasks is rather difficult. Providing expert demonstration is often easier than designing effective reward functions. Thus, we propose a model-free stable adversarial inverse reinforcement learning (SAIRL) algorithm that only adopts the state of MSV and reconstructs the reward function from the expert demonstration. The SAIRL algorithm is designed to guarantee the prescribed MSV path following accuracy and training stability. It utilizes an alternative loss function and dual-discriminator framework to dissolve the issue of policy collapse, which arises due to the vanishing gradient of the discriminator. Simulations and experiments have validated that the SAIRL algorithm outperforms other baseline algorithms in terms of path-following accuracy and stability of convergence.

Reconstructing human gaze behavior from EEG using inverse reinforcement learning

Decoding eye movements from non-invasive electroencephalography (EEG) data is a challenging yet vital task for both scientific and practical purposes, especially for identifying neurodegenerative disorders like Alzheimer’s disease (AD). Our research tackles this complexity by adapting inverse reinforcement learning (IRL), a machine learning method, to infer decision-making strategies from observed behaviors. We implement this to understand the processes driving eye direction and movements during diverse cognitive tasks, providing new insights into this field. Our paper begins with a detailed description of the procedures for collecting and preprocessing EEG data related to gaze behavior. We then elaborate on the development of an IRL framework designed to predict the spatial and temporal dynamics of eye movements (scanpaths) in participants engaged in cognitive tasks of varying complexity. Our model is tailored to accommodate the complexities inherent in neural signals and the stochastic nature of human gaze patterns. Our research findings underscore IRL’s effectiveness in precisely forecasting gaze patterns based on a combination of EEG and image data. The correlation between the model’s predictions and the actual gaze behavior observed in controlled experiments reinforces the utility of IRL in cognitive neuroscience research. Notably, our IRL-EEG models demonstrated superior performance, especially in more complex cognitive tasks. We further delve into the implications of our results for enhancing the understanding of neural mechanisms that govern gaze behavior.

Inverse Reinforcement Learning for Trajectory Imitation Using Static Output Feedback Control.

This article studies the trajectory imitation control problem of linear systems suffering external disturbances and develops a data-driven static output feedback (OPFB) control-based inverse reinforcement learning (RL) approach. An Expert-Learner structure is considered where the learner aims to imitate expert's trajectory. Using only measured expert's and learner's own input and output data, the learner computes the policy of the expert by reconstructing its unknown value function weights and thus, imitates its optimally operating trajectory. Three static OPFB inverse RL algorithms are proposed. The first algorithm is a model-based scheme and serves as basis. The second algorithm is a data-driven method using input-state data. The third algorithm is a data-driven method using only input-output data. The stability, convergence, optimality, and robustness are well analyzed. Finally, simulation experiments are conducted to verify the proposed algorithms.

Bridging the Gap Between Self-Report and Behavioral Laboratory Measures: A Real-Time Driving Task With Inverse Reinforcement Learning

A major challenge in assessing psychological constructs such as impulsivity is the weak correlation between self-report and behavioral task measures that are supposed to assess the same construct. To address this issue, we developed a real-time driving task called the “highway task,” in which participants often exhibit impulsive behaviors mirroring real-life impulsive traits captured by self-report questionnaires. Here, we show that a self-report measure of impulsivity is highly correlated with performance in the highway task but not with traditional behavioral task measures of impulsivity (47 adults aged 18–33 years). By integrating deep neural networks with an inverse reinforcement learning (IRL) algorithm, we inferred dynamic changes of subjective rewards during the highway task. The results indicated that impulsive participants attribute high subjective rewards to irrational or risky situations. Overall, our results suggest that using real-time tasks combined with IRL can help reconcile the discrepancy between self-report and behavioral task measures of psychological constructs.